The efficacy of the Sortino ratio and other benchmarked performance measures under skewed return distributions

This paper will investigate the suitability of existing performance measures under the assumption of a clearly defined benchmark. A range of measures are examined including the Sortino Ratio, the Sharpe Selection ratio (SSR), the Student’s t-test and a decay rate measure. A simulation study is used to assess the power and bias of these measures based on variations in sample size and mean performance of two simulated funds. The Sortino Ratio is found to be the superior performance measure exhibiting more power and less bias than the SSR when the distribution of excess returns are skewed.

Dynamic Performance Measures for Object Tracking Systems

Performance evaluation of object tracking systems is typically performed after the data has been processed, by comparing tracking results to ground truth. Whilst this approach is fine when performing offline testing, it does not allow for real-time analysis of the systems performance, which may be of use for live systems to either automatically tune the system or report reliability. In this paper, we propose three metrics that can be used to dynamically asses the performance of an object tracking system. Outputs and results from various stages in the tracking system are used to obtain measures that indicate the performance of motion segmentation, object detection and object matching. The proposed dynamic metrics are shown to accurately indicate tracking errors when visually comparing metric results to tracking output, and are shown to display similar trends to the ETISEO metrics when comparing different tracking configurations.

Performance measures for Australian laboratories reporting cervical cytology : a decade of data 1998–2008

Aim Performance measures for Australian laboratories reporting cervical cytology are a set of quantifiable measures relating to the profile and accuracy of reporting. This study reviews aggregate data collected over the ten years in which participation in the performance measures has been mandatory. Methods Laboratories submit annual data on performance measures relating to the profile of reporting, including reporting rates for technically unsatisfactory specimens, high grade or possible high grade abnormalities and abnormal reports. Cytology-histology correlation data and review findings of negative smears reported from women with histological high grade disease are also collected. Suggested acceptable standards are set for each measure. This study reviews the aggregate data submitted by all laboratories for the years 1998-2008 and examines trends in reporting and the performance of laboratories against the suggested standards. Results The performance of Australian laboratories has shown continued improvement over the study period. There has been a fall in the proportion of laboratories with data outside the acceptable standard range in all performance measures. Laboratories are reporting a greater proportion of specimens as definite or possible high grade abnormality. This is partly attributable to an increase in the proportion of abnormal results classified as high grade or possible high grade abnormality. Despite this, the positive predictive value for high grade and possible high grade abnormalities has continued to rise. Conclusion Performance measures for cervical cytology have provided a valuable addition to external quality assurance procedures in Australia. They have documented continued improvements in the aggregate performance, as well as providing benchmarking data and goals for acceptable performance for individual laboratories.

Beyond words: Testing alignment among inter-organizational performance measures

Purpose: This paper seeks to measure in a quantitative way the degree of alignment among a set of performance measures between two organizations. Design/methodology/approach: This paper extends Venkatraman's test of coalignment to assess the alignment of a set of performance measures governing a contractual inter-organizational relationship. The authors applied the test and present coefficients of misalignment across three sets of measures: those used by a service provider involved in the research, those used by customers contracting the services, and those documented in 11 contracts studied. Findings: Results confirmed a high degree of alignment between target and actual operational performance in the contracts. The alignment of customers' financial objectives and contracts' operational metrics was low. Calculations show poor alignment between the objectives of the provider and the contribution received from the contracts. Research limitations/implications: Some limitations of the conclusions include the small sample of contracts used in the calculations. Further research should include not only actual contracts, but also failed ones. Practical implications: It is possible that misaligned goals, represented in misaligned performance measures, lead to tensions in intra-firm relationships. If these tensions are not addressed properly the relationship could be unstable or terminated prematurely. This method of measuring alignment could detect early potential dangers in intra-firm relationships. Originality/value: This paper extends Venkatraman's test of coalignment to assess the alignment of a set of performance measures governing a contractual inter-organizational relationship. Management researchers and business professionals may use this methodology when exploring degrees of alignment of performance measures in intra-functional and inter-firm relationships. © Emerald Group Publishing Limited.

Comparative Performance Measures of Fuzzy ARTMAP, Learned Vector Quantization, and Back Propagation for Handwritten Character Recognition

This article compares the performance of Fuzzy ARTMAP with that of Learned Vector Quantization and Back Propagation on a handwritten character recognition task. Training with Fuzzy ARTMAP to a fixed criterion used many fewer epochs. Voting with Fuzzy ARTMAP yielded the highest recognition rates.

Statistical Tests for Joint Analysis of Performance Measures

Recently there has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and architectural complexity). Once one has learned a model based on their devised method, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Unfortunately, the standard tests used for this purpose are not able to jointly consider performance measures. The aim of this paper is to resolve this issue by developing statistical procedures that are able to account for multiple competing measures at the same time. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameter of such models, as usually the number of studied cases is very reduced in such comparisons. Real data from a comparison among general purpose classifiers is used to show a practical application of our tests.

Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures

Many different performance measures have been developed to evaluate field predictions in meteorology. However, a researcher or practitioner encountering a new or unfamiliar measure may have difficulty in interpreting its results, which may lead to them avoiding new measures and relying on those that are familiar. In the context of evaluating forecasts of extreme events for hydrological applications, this article aims to promote the use of a range of performance measures. Some of the types of performance measures that are introduced in order to demonstrate a six-step approach to tackle a new measure. Using the example of the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble precipitation predictions for the Danube floods of July and August 2002, to show how to use new performance measures with this approach and the way to choose between different performance measures based on their suitability for the task at hand is shown. Copyright © 2008 Royal Meteorological Society

Bayesian Estimation for Performance Measures of Two Diagnostic Tests in the Presence of Verification Bias

Sensitivity and specificity are measures that allow us to evaluate the performance of a diagnostic test. In practice, it is common to have situations where a proportion of selected individuals cannot have the real state of the disease verified, since the verification could be an invasive procedure, as occurs with biopsy. This happens, as a special case, in the diagnosis of prostate cancer, or in any other situation related to risks, that is, not practicable, nor ethical, or in situations with high cost. For this case, it is common to use diagnostic tests based only on the information of verified individuals. This procedure can lead to biased results or workup bias. In this paper, we introduce a Bayesian approach to estimate the sensitivity and the specificity for two diagnostic tests considering verified and unverified individuals, a result that generalizes the usual situation based on only one diagnostic test.